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DETAILED ACTION 

Specification 

1 . The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. 

The following title is suggested: Interactive and Shared Augmented Reality 
System and Method having Local and Remote Access. 



Claim Objections 

1 . Claims 10, 12, 24, 33 objected to because of the following informalities. 
Appropriate correction is required. 

As to claim 10, it fails to depend on a previous claim in which claim 1 1 
does not precede claim 10. It is suggested to make claim 10 depend on 10, 
renumber claim 10 to 1 1 and claim 1 1 to 10. 

As to claim 12, it is suggested to change the phrase "A The system" to 
"The system." 

As to claim 24, it is suggested to change the phrase "being arranged in a 
fix relation" to "being arranged in a fixed relation." Also there is no antecedent 
basis for "the local display device." It is suggested to remove the antecedence 
by changing it to "a local display device." 

As to claim 33, it is suggested to change the phrase "Use of the method" 
to "Use of the system" as it depends on claim 1 . 
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Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. . Claims 1-29, 33 rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent Application No. 2002/0010734 A1 of Ebersole et al. ("Ebersole") in view of 
U.S. Patent Application No. 2002/0082498 A1 of Wendt et al. ("Wendt"). 

As to claim 1, Ebersole discloses an augmented reality system (Title) 
comprising: a camera 34 for capturing an image, the camera being movably 
located at a local site (fig. 3, page 2 [0030] lines 1-2, 15-18, page 5 [0059] - local 
AR station viewpoint is transmitted to a Remote Station such as a Remote AR 
Station in which the user is remotely located which requires a video see-through 
AR system), a registering unit 32, generating graphics and registering the 
generated graphics to the image from the camera, to provide a composite 
augmented reality image (figs. 5-6), a display device 32 located at a remote site, 
physically separated from the local site (page 2 [0025] lines 10-13), for displaying 
a view comprising the composite augmented reality image (fig. 3, page 2 [0028] 
lines 2-4 - AR display system), and a communication link 2, for communication of 
information between the local and the remote site (fig. 1, page 3 [0032]), wherein 
the system further comprises a specifying unit 33, for specification of a position 
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and an orientation in the remote site (fig. 2, 3, page 2 [0029] lines 1-2, page 5 
[0059] - considering a Remote AR Station 1 , 6 in which the head worn display 
specifies a position and orientation), and the registering unit is adapted for 
registering the generated graphics to the image in dependence of the specified 
position and orientation (page 3 [0034] lines 15-22). 

Ebersole does not expressly disclose the camera being movably located 
at a local site; the camera is arranged such that its position and orientation is 
dependent on the specified position and orientation. He does not fully disclose 
the implementation of a Remote AR Station 1, 6 (fig. 2, 3), specifically how the 
Remote AR Station 1 , 6 obtains its view in the Local AR Station 3 i.e. through a 
camera on a another person, or a remote controlled robotic arm or vehicle. 

Wendt discloses an augmented reality system comprising: a camera 64 
for capturing an image, the camera being movably located at a local site (fig. 6, 
page 4 [0056] lines 1-2), a display device 66 located at a remote site (fig. 6), the 
camera is arranged such that its position and orientation is dependent on the 
specified position and orientation (page 4 [0055-0057]). 

Ebersole and Wendt are analogous art because they are from the same 
field of endeavor namely augmented reality systems. 

At the time of the invention it would have been obvious to one of ordinary 
skill in the art to modify the previous Remote AR Station to obtain its view in the 
Local AR Station by means of a camera mounted on a robotic arm located in the 
Local AR Station as taught above by Wendt. The motivation would have been to 
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enable independent movement and operation of the remote user in the Local AR 
Station (Wendt page 4 [0057]). 

Therefore it would have been obvious to combine Ebensole and Wendt to 
obtain the above modifications. 

As to claim 2, Ebersole discloses the system according to claim 1, 
wherein said specifying unit 33 comprises a tracking unit 47, adapted for 
determining the position and orientation of a movable device 45 (page 2 [0029] 
lines 1-2) located at the remote site (fig. 3, 4, page 5 [0059] - considering a 
Remote AR Station 6), the registering unit is adapted for registering the 
generated graphics to the image in dependence of the position and orientation of 
the movable device, and the camera is arranged such that its position and 
orientation are dependent on the position and orientation of the movable device. 

As to claim 3, Ebersole discloses the system according to claim 2, 
wherein said movable device is the display device 45. 

As to claim 4, Wendt discloses the system according to claim 1 , further 
comprising a robot 62 located at the local site (fig. 6), the camera is mounted on 
the robot 62 and the robot is arranged in such a manner that the movement of 
the robot depends on the specified position and orientation (page 4 [0057]). 

The same motivation is used here as is used in the parent claim. 

As to claim 5, Ebersole discloses the system according to claim 1 , further 
comprising a graphical generator, for generation of a graphical representation, 
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and the registering unit is adapted for generating graphics based on the graphical 
representation (figs. 6-8, 10, page 2 [0028]). 

As to claim 6, Ebersole discloses the system according to claim 1 , further 
comprising operator input means 35, located at the remote site (considering the 
Remote AR Station), provided for feeding data related to the graphics to be 
displayed to the system, and the system is adapted for generating the graphics 
based on said data (fig. 3, 4, 6, page [0039]). 

As to claim 7, Ebersole discloses the system according to claim 6, 
wherein said operator input means comprises a pointing device and a tracking 
unit for determining the position of the pointing device and that the system is 
adapted for generating a graphical representation of a point being presently 
pointed out by the pointing member based on the position of the pointing device 
(page 2 [0031] , page 4 [0050] lines 16+ - for example a mouse is a pointing 
device that has tracking means, wherein further when placing an object at a 
specific point that point has a graphical representation). 

As to claim 8, Ebersole discloses the system according to claim 1, further 
comprising a second specifying unit 33, for specifying a position and an 
orientation in the local site (fig. 2, 3, page 2 [0029] lines 1-2 - considering a 
second Local AR station 4, 6), a second registering unit 32, generating graphics 
and registering the generated graphics to the real environment or an image of the 
environment of the local site (page 3 [0034] lines 15-22), in dependence of the 
position and orientation specified by the second specifying unit, and a local 
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display device 32 adapted for displaying a view comprising the environment of 
the local site and said generated graphics projected on the environment (fig. 3, 
page 2 [0028] lines 2-4). 

As to claim 9, Ebersole discloses the system according to claim 8, further 
comprising a second movable device 47 located at the local site 4, 6, the second 
specifying unit comprises a second tracking unit 47, for determining the position 
and the orientation of the second movable device 45 (figs. 2-4 - considering a 
second Local AR Station 4, 6). 

As to claim 10, Ebersole discloses the system according to claim 1 1 , 
wherein said second movable device 47 is the local display device 47. 

As to claim 11, Ebersole discloses the system according claim 9, further 
comprising a second camera 34a for capturing an image, the camera 34a being 
arranged in a fix relation to the second movable device 45, and the second 
registering unit, is adapted for registering the generated graphics to the image 
from the second camera, to provide a composite augmented reality image, and 
that the local display device 45 is adapted for displaying a view comprising the 
composite augmented reality image (fig. 3, 4, page 2 [0030] lines 1-2, 15-18 - a 
video see-through device as disclosed). 

As to claim 12, Ebersole discloses the system according to claim 8, 
wherein the remote display device is adapted for displaying a view seen from a 
first visual angle that depends on the position and orientation received from the 
first mentioned specifying unit and the local display device is adapted for 
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displaying the same view as the remote display device seen from a second visual 
angle that depends on the position and orientation received from the second 
specifying unit (Title, Abstract, page 5 [0057] - internetworked AR). 

As to claim 13, Ebersole discloses the system according to claim 1, 
further comprising means for transferring voices between the remote and the 
local site via the communication link 2 (page 2 [0025] lines 15-16, page 6 [0060]). 

As to claim 14, Ebersole discloses the system according to claim 1 , 
wherein the communication link is a network (fig. 1, page 3 [0032]). 

As to claim 15, Ebersole discloses a method for remotely displaying an 
augmented reality view comprising graphical information overlaid an image 
captured at a local site (figs. 6, 8-10), the method comprising: specifying a 
position and an orientation at a remote site (fig. 2, 3, page 2 [0029] lines 1-2, 
page 5 [0059] - considering a remote AR station 1 , 6 in which the head worn 
display specifies a position and orientation) that is physically separated from the 
local site (page [0025] lines 10-13), obtaining an image from the camera, 
generating graphics, generating a composite augmented reality image based on 
the image, the graphics, and the specified position and orientation, and 
displaying a view comprising the composite augmented reality image (fig. 3, 5, 6, 
7, 10, page 2 [0028]). 

Ebersole does not expressly disclose positioning and orientating a 
camera, located at the local site, according to the specified position and 
orientation. 
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Wendt discloses positioning and orientating a camera 64, located at the 
local site (fig. 6, page 4 [0056] lines 1-2), according to the specified position and 
orientation (page 4 [0055-0057]). 

Ebersole and Wendt are analogous art because they are from the same 
field of endeavor namely augmented reality systems. 

At the time of the invention it would have been obvious to one of ordinary 
skill in the art to modify the previous Remote AR Station 1 , 6 camera 34 to be 
located and moveable in the Local AR Station 3 (Ebersole fig. 2, 3) as taught 
above by Wendt. The motivation would have been to enable independent 
movement and operation of the remote user in the Local AR Station (Wendt page 
4 [0057]). 

Therefore it would have been obvious to combine Ebensole and Wendt to 
obtain the above modifications. 

As to claim 16, Ebersole discloses the method according to claim 15, 
wherein specifying a position and an orientation comprises determining the 
position and orientation of a movable device 45 (fDage 2 [0029] lines 1-2) located 
at the remote site (fig. 3, 4, page 5 [0059] - considering a Remote AR Station 6) 
and the camera is positioned and orientated according to the position and 
orientation of the movable device. 

As to claim 17, Ebersole discloses the method according to claim 16, 
wherein said movable device is a remote display device and that said view 
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comprising the composite augmented reality image is displayed on the remote 
display device (fig. 3, page 2 [0028] lines 2-4 - AR display system). 

As to claim 18, Wendt discloses the method according to claim 15, further 
comprising controlling the movements of a robot 62 (page 4 [0056-0058]), having 
the camera 64 mounted thereon (fig. 6), according to the position and orientation 
of the movable device (page 4 [0057]). 

The same motivation is used here as is used for the parent claim. 

As to claim 19, Ebersole discloses the method according to claim 15, 
further comprising obtaining data related to the graphics to be displayed, and 
generating the graphics based on said data (fig. 3, 4, 6, page [0039]). 

As to claim 20, Ebersole discloses the method according to claim 15, 
further comprising receiving information about the position of a pointing device 
and generating graphics representing a point, being presently pointed out by the 
pointing member, based on the position of the pointing device (page 2 [0031], 
page 4 [0050] lines 16+ - for example a mouse is a pointing device that has an 
identifiable position wherein generated virtual objects or graphics may be placed 
at the identified position to represent the identified position). 

As to claim 21, Ebersole discloses the method according to claim 15, 
further comprising specifying a position and an orientation in the local site (fig. 2, 
3 - considering a second Local AR Station 4, 6 using the tracking unit 33), and 
displaying a second view comprising the environment of the local site and the 
generated graphics projected on the environment in dependence of the locally 
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specified position and orientation (fig. 3, page 2 [0028] lines 2-4, page 3 [0034] 
lines 15-22). 

As to claim 22, Ebersole discloses the method according to claim 21 , 
wherein specifying a position and an orientation in the local site comprises 
determining the position and orientation of a second movable device 45 located 
at the local site (figs. 2-4, page 2 [0029] - considering a second Local AR Station 
4,6). 

As to claim 23, Ebersole discloses the method according to claim 22, 
wherein the second movable device 45 is a local display device 45 and that said 
second view, comprising the environment of the local site and the graphics, is 
displayed on the local display device (figs. 2-4, page 2 [0029] - considering a 
second Local AR Station 4, 6). 

As to claim 24, Ebersole discloses the method according to claim 22, 
further comprising capturing an image from a second camera 34a being arranged 
in a fix relation to the second movable device 45 (fig. 4), and registering the 
generated graphics to the image from the second camera, to provide a 
composite augmented reality image, and displaying a view comprising the 
composite augmented reality image on the local display device 45 (fig. 3, 4, page 
2 [0030] lines 1-2, 15-18 - a video see-through device as disclosed). 

As to claim 25, Ebersole discloses the method according to claim 21 , 
further comprising generating second graphics and displaying the second view 
comprising the environment of the local site and the second graphics projected 
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on the environment in dependence of the specified position and orientation (fig. 
3, page 2 [0028] lines 2-4, page 3 [0034] lines 15-22, 37-38, page 4 [0040] - an 
instructor do not have to be at the Local AR Station and often are at Remote 
Non-AR Stations and may be at a Remote AR Station and even Local AR 
Stations; they may also see a first-person view of any trainee in the AR). 

As to claim 26, Ebbersole discloses the method according to claim 25, 
further comprising generating a local graphical representation, generating a 
remote graphical representation, transferring the local and remote graphical 
representations between the local and the remote site (fig. 1, page 3 [0032]), 
generating the first mentioned graphics based on the local and the remote 
graphical representation, and generating the second graphics based on the local 
and the remote graphical representation (page 3 [0037], page 4 [0042-0047], 
page 5 [0058-0060] - the collaboration of textual messages involves local and 
remote graphical representations in the form of text wherein when displayed they 
become a part of the local and remote graphics). 

As to claim 27, Ebersole discloses the method according to claim 21, 
wherein the view displayed in the remote site comprises the environment of the 
local site and the overlaid graphics seen from an visual angle that depends on 
the position and orientation specified in the remote site and the view displayed in 
the local site comprises the environment of the local site and the overlaid 
graphics seen from an visual angle that depends on the position and orientation 
specified in the local site (Title, Abstract, page 5 [0057] - internetworked AR). 
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As to claim 28, Ebersole discloses a computer program product directly 
loadable into the internal memory of a computer, comprising software code 
portions for performing the steps of claim 1 5 (Appendix A), when said product is 
run on a computer 31 (fig. 3, page [0027] - for the AR system of multiple stations 
having computing systems 31 being PCs or better, they all must load the 
program of Appendix A into internal memory in order to run). 

As to claim 29, Ebersole discloses a computer readable medium having a 
program recorded thereon, where the program is to make a computer perform 
the steps of claim 15 (Appendix A), when said program is run on the computer 
(fig. 2, 3, page [0027] - for the AR system of multiple stations having computing 
systems 31 being PCs or better, they all have a computer readable medium that 
stores the program of Appendix A such as a hard drive). 

As to claim 33, Ebersole discloses use of the method according to claim 1 
for a paint application (fig. 10, page 5 [0051-0053] - The application of color and 
texture to objects embody a paint application in the AR). 

3. Claim 30 rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 

Patent Application No. 2002/0010734 A1 of Ebersole et al. ("Ebersole") in view of U.S. 

Patent Application No. 2002/0082498 A1 of Wendt et al. ("Wendt") as applied to claim 1 

above, and further in view of U.S. Patent No. 5,745,387 A to Corby, Jr. et al. ("Corby") 
As to claim 30, it has not expressly been disclosed yet. 
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Corby discloses use of a system according to claim 1 for remote 
programming (col. 5 lines 29-53 - the remote operator may program intended 
paths of robot arm) of an industrial robot 10-14. 

Ebersole and Corby are analogous art because they are from the same 
field of endeavor namely augmented reality systems. 

At the time of the invention it would have been obvious to one of ordinary 
skill in the art to modify the previous robot arm to be automatic and 
programmable as taught above by Corby. The motivation would have been to 
accurately and automatically position the robot arm (Corby col. 3 lines 34-40). 

Therefore it would have been obvious to combine Ebersole, Wendt and 
Corby to obtain the above modifications. 
4. Claims 31-32 rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent Application No. 2002/0010734 A1 of Ebersole et al. ("Ebersole") in view of U.S. 
Patent Application No. 2002/0082498 A1 of Wendt et al. ("Wendt") as applied to claim 1 
above, and further in view of U.S. Patent No. 5,815,41 1 A to Ellenby et al. ("Ellenby"). 

As to claim 31 , it has not yet expressly been disclosed. 

Ellenby discloses the system according to claim 1 1 , further comprising a 
handheld display device (fig. 1 ) comprising the display member 13 and the 
camera 9 (figs. 1-4). 

Ebersole and Ellenby are analogous art because they are from the same 
field of endeavor namely augmented reality systems. 
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At the time of the invention it would have been obvious to one of ordinary 
skill in the art to modify the previous AR display system to instead be a handheld 
tracked display like binoculars as taught above by Ellenby. The motivation would 
have been to allow the user to more naturally view the AR (Ellenby col. 3 line 63) 
as they normally would (Ellenby col. 9 lines 47-51 ) wherein certain situations a 
handheld device would be more natural to use then a head mountable display. 

Therefore it would have been obvious to combine Ebersole, Wendt and 
Ellenby to obtain the above modifications. 

As to claim 32, Ellenby further discloses the system according to claim 
31 , wherein the handheld display device is arranged so that the user seems to 
look directly through the display (Claim 4, col. 9 lines 47-51). 

The same motivation is used here as is used in the parent claim. 

Conclusion 

5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

As to claim 11, U.S. Patent Application 2003/0003430 A1 of Ebersole et 
al. discloses a tracked pointing device. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Paul Saunders whose telephone number is 
571.270.3319. The examiner can normally be reached on Mon-Thur 8:30am-4:00pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Derrick Ferris can be reached on 571.272.3123. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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